Simultaneous capture of filtered images of the eye

ABSTRACT

A multimode fundus camera enables three-dimensional and/or spectral/polarization imaging of the interior of the eye to assist in improved diagnosis.

BACKGROUND OF THE INVENTION 1. Field of the Invention

This invention relates generally to imaging of the eye, for examplespectral, polarization and/or three-dimensional imaging of the retina.

2. Description of the Related Art

Specialized cameras are used by optometrists, ophthalmologists, andother medical professionals to record images of the interior surface ofthe eye. During a routine physical examination, a handheldophthalmoscope is often used to quickly view the fundus. Additionalcamera attachments can be used to record digital images from handheldophthalmoscopes, allowing acquired images to be saved, manipulated, andreevaluated at future examinations. However, these images are limited bybrightness, field-of-view, motion blur, and resolution, which restricttheir diagnostic abilities for many diseases.

More complex imaging systems (e.g., fundus camera) can provide theclinician with better image quality, leading to more accurate diagnosis,screening, and monitoring treatment of eye pathologies. Conventionalfundus cameras provide an image of the fundus with 2 to 5×magnification, with a field-of-view of 15 to 140 degrees. The devicetypically incorporates specialized illumination optics to shine lightonto the interior surface of the eye. An eyepiece can be used to allowthe clinician to view the interior of the eye. An electronic sensor canbe used for digital acquisition of images. During an examination, themedical professional inspects the interior of the eye for abnormalitiessuch as retinal tearing, thinning, unhealthy vasculature, opacity,occlusions, enlarged or reduced anatomy, and discoloration.

However, conventional fundus cameras have several drawbacks. First, inmany cases, absolute measurements of anatomical features would bebeneficial to determine the type and severity of disease. However,conventional fundus cameras produce a two-dimensional image of thethree-dimensional eye. This makes it difficult or impossible to assessabsolute measurements of area, depth or volume for the three-dimensionalanatomy. Second, in many cases, spectral, polarization or other imagingmodalities would also be beneficial. Conventional fundus camerastypically might capture different filtered images sequentially in time.Snapshots taken at different times must then be registered with eachother. However, since the eye is constantly moving, this introduces aregistration problem. In a different approach, it is possible to modifyconventional fundus cameras to capture multiple filtered images in asingle snapshot, for example by using multiple sensor arrays. However,this makes the camera more complex and expensive, and the multipleoptical paths must be aligned to each other to ensure correct imageregistration.

Therefore, there exists a need for improved imaging systems to allow thesimultaneous capture of three-dimensional, spectral, polarization andother modality images.

SUMMARY OF THE INVENTION

In one aspect, a multimode imaging system includes a first imagingsubsystem, a filter module and a second imaging subsystem. The firstimaging subsystem includes an objective lens, which is positionable infront of an eye to form an optical image of an interior of the eye(e.g., of the retina of the eye). The filter module, which includesmultiple filters, is positioned at a pupil plane of the first imagingsubsystem. The second imaging subsystem includes a microimaging arrayand a sensor array. The microimaging array (e.g., a microlens array) ispositioned at the image plane of the first imaging subsystem, and thesensor array is positioned at a conjugate of the pupil plane. The sensorarray captures a plenoptic image of the interior of the eye, whichcontains images of the interior of the eye filtered by each of theplurality of filters. In alternate embodiments, the system may includerelay optics, allowing components to be positioned at conjugates of theimage plane and pupil plane respectively.

Another aspect is an after-market conversion kit for converting aconventional fundus camera to a multimode imaging system as describedabove.

Other aspects include methods, devices, systems, and applicationsrelated to the approaches described above and its variants.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The invention has other advantages and features which will be morereadily apparent from the following detailed description and theappended claims, when taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 (prior art) are example images showing different conditions ofthe eye.

FIGS. 2a-b are diagrams illustrating an example of a multimode imagingsystem.

FIG. 3 is a diagram of an example of another multimode imaging system.

The figures depict embodiments for purposes of illustration only. Oneskilled in the art will readily recognize from the following discussionthat alternative embodiments of the structures and methods illustratedherein may be employed without departing from the principles describedherein.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The figures and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed. To facilitate understanding, identical referencenumerals have been used where possible, to designate identical elementsthat are common to the figures.

FIG. 1 (prior art) are example images showing different conditions ofthe eye. The images are images of the retina. The entries below eachretina image list the disease condition, the features that can be usedto diagnose the disease, and the modalities that would be useful toimage the features. The leftmost column is a healthy eye.

The first disease listed is glaucoma, which has a clinical presentationof an enlarged optic disc. In order to assess enlargement of the opticdisc, a medical professional might attempt to estimate the cup-to-discratio, as indicated in the second row of the table. He might do this bycomparing current images of the retina to images from previous exams, bycomparing optic disc symmetry between eyes, or by looking for thinningof the disc rim. In a conventional fundus camera, these roughmeasurements of a three-dimensional eye are estimated from atwo-dimensional image. The analysis would be more accurate if they wereestimated from a three-dimensional image instead, as indicated by the 3Dmodality listed in the third row of the table. In addition, the opticdisc in a patient with glaucoma can also respond differently topolarized light. Increased intraocular pressure, retinal thinning, andchanges in the optic disc can change birefringence properties or causeother polarization-related effects. These changes can be assessed bypolarization images, as indicated by the polarization modality in thethird row of the table. Conventional fundus cameras typically do notprovide such polarization measurements.

Eye diseases can also change the vascular structure and physiologicalactivity of the tissue, which alters the metabolism of tissue areas. Forexample, the second disease listed in FIG. 1 is diabetic retinopathy.This disease is classified by the type and severity of lesions,including: microaneurysms, hemorrhages, cotton wool spots, and venousbeading. Three-dimensional measurement of these lesions could helpobjectively assess the severity of disease. Additionally, spectralimaging can indicate health of the retinal tissue by use of narrowbandfilters. In this approach, two or more narrowband spectral filters areselected based on the known spectral response of oxy-hemoglobin. Imagesacquired with these spectral filters are then used to generate an oxygensaturation map of the tissue. This oximetry map provides an additionalclinical measurement of the tissue, which can greatly aid in diagnosis.Conventional fundus cameras do not provide such measurements.

The diagnosis of the third and fourth diseases listed in FIG. 1 wouldsimilarly be improved by three-dimensional, spectral, and/orpolarization measurements.

FIGS. 2a-b are diagrams illustrating an example of a multimode imagingsystem. The imaging system 210 includes an objective lens 212(represented by a single lens in FIG. 2a ), a secondary imaging array214 (an array of image forming elements) and a sensor array 280. Forconvenience, the imaging optics 212 is depicted in FIG. 2a as a singleoptical element, but it should be understood that it could containmultiple elements.

The secondary imaging array 214 may be referred to as a microimagingarray. The secondary imaging array 214 and sensor array 280 together maybe referred to as a plenoptic sensor module. In this example, thesecondary imaging array 214 is a microlens array. Other examples ofmicroimaging arrays 214 include microlens arrays, arrays of pinholes,micromirror arrays, checkerboard grids and waveguide/channel arrays. Themicroimaging array 214 can be a rectangular array, hexagonal array orother types of arrays.

These components form two overlapping imaging subsystems. In the firstimaging subsystem, the objective lens 212 is positionable in front ofthe eye 250 and forms an optical image 255 of the eye (retina, in thisexample) at the primary image plane IP, which may be relayed toconjugate planes such as the image port IP′. This imaging subsystem hasa pupil plane. In the second imaging subsystem, the secondary imagingarray 214 images the pupil plane onto the sensor array 280. To do this,the microimaging array 214 is located at the image plane IP or one ofits conjugate planes. In this example, the microlens array 214 islocated at conjugate plane IP′. The system in its entirety formsspatially multiplexed and interleaved optical images 270 at the sensorplane SP.

A filter module 225 is positioned at a plane SP′ conjugate to the sensorplane SP. The actual physical location may be before, after or in themiddle of the imaging optics 212. The filter module contains a number ofspatially multiplexed filters 227A-D. In this example, the filter module225 includes a rectangular array of filters 227, as shown in the bottomportion of FIG. 2a . The filter module 225 could contain spectralfilters, polarization filters, neutral density filters, clear filters(i.e., no filters) or combinations of these.

The top portion of FIG. 2a provides more detail. In this diagram, theretina 250 is divided into a 3×3 array of regions, which are labeled1-9. The filter module 225 is a 2×2 rectangular array of individualfilters 227A-D. For example, each filter 227A-D may have a differentspectral response. The sensor array 280 is shown as a 6×6 rectangulararray.

FIG. 2b illustrates conceptually how the spatially multiplexed opticalimages 270A-D are produced and interleaved at sensor array 280. Theobject 250, if captured and filtered by filter 227A, would produce anoptical image 255A. To distinguish filtered optical image 255A from anunfiltered image of the object, the 3×3 regions are labeled with thesuffix A: 1A-9A. Similarly, the object 250 filtered by filters 227B,C,D,would produce corresponding optical images 255B,C,D with 3×3 regionslabeled 1B-9B, 1C-9C and 1D-9D. Each of these four optical images 255A-Dis filtered by a different filter 227A-D within filter module 225 butthey are all produced simultaneously by the imaging system 210. Thisallows different modality images to be captured in a single snapshot,eliminating the need to later compensate for eye movement whenregistering images.

The four optical images 255A-D are formed in an interleaved fashion atthe sensor plane, as shown in FIG. 2B. Using image 255A as an example,the 3×3 regions 1A-9A from optical image 255A are not contiguous in a3×3 block within optical image 270. Rather, regions 1A, 1B, 1C and 1D,from the four different optical images, are arranged in a 2×2 fashion inthe upper left of optical image 270 (the inversion of image 270 isneglected for clarity). Regions 1-9 are similarly arranged. Thus, theregions 1A-9A that make up optical image 270A are spread out across thecomposite optical image 270, separated by portions of the other opticalimages 270B-D. Put in another way, if the sensor is a rectangular arrayof individual sensor elements, the overall array can be divided intorectangular subarrays 271(1)-(9) of sensor elements (only one subarray271(1) is shown in FIG. 2B). For each region 1-9, all of thecorresponding regions from each filtered image are imaged onto thesubarray. For example, regions 1A, 1B, 1C and 1D are all imaged ontosubarray 271(1). Note that since the filter module 225 and sensor array280 are located in conjugate planes, each imaging element in array 214forms an image of the filter module 225 at the sensor plane SP. Sincethere are multiple imaging elements, multiple images 271 of the filtermodule 225 are formed.

The multiplexed image 270 can be processed by processing module 290 toreconstruct desired images of the object. The processing could bedeinterleaving and demultiplexing. It could also include moresophisticated image processing. In addition to experiencing differentfiltering, the image data captured by system 210 also reflects differentviewpoints. That is, the multiplexed images are captured from differentviewpoints. This information can be used to reconstruct athree-dimensional image of the retina or to reduce the effects ofocclusions. Thus, the reconstructed images 295 can includethree-dimensional information in addition to filtered images (e.g.,color and/or polarization images). The system could be designed so thatit is switchable between a depth mode and a multi-filter mode.Alternately, the system can capture both depth and spectral/polarizationinformation simultaneously.

For example, oxygen saturation of the retina can help predict diabeticretinopathy. Retinal oximetry can be measured with two or more spectralfilters: one or more filters are selected at wavelengths whereoxy/deoxy-hemoglobin spectra are most separated (such as anywherebetween 600-700 nm), and one at an isosbestic point (such as 586 nm or808 nm). For example, the system might use one filter centered at 548 nmwith a 10 nm width (close to isosbestic) and a second filter centered at610 nm with a 10 nm width. Additionally, near infrared wavelengths canbe used to increase contrast of vasculature, especially deep vessels.

Regarding polarization, in practice, the illumination light may bepolarized and tuned to account for birefringence of the cornea. For theanalysis of reflected light, unpolarized, linear, and circular polarizedfilters can be used to assess polarization-dependent properties of theretina. Rods and cones maintain the polarization of incident light. In ahealthy retina, the optic disc has no rods/cones and therefore causesdepolarization of reflected light. Disease associated with malformedanatomy or increased intraocular pressure (glaucoma) can alterbirefringence properties and polarization response in those areas.

Depth information can be used to more accurately estimate the size ofthe optic disc for glaucoma diagnosis. Without consideration of theoptic disc size, the cup size and cup/disc ratio are not clinicallymeaningful. Current methods are typically qualitative or relativemeasurements. However, absolute measurement of the optic disc provides aquantitative feature for disease classification. Three-dimensionalinformation may also be used to identify or measure physicaldeformations.

It should be noted that FIG. 2 has been simplified to illustrateunderlying concepts. For example, the object 250 was artificiallydivided into an array in order to more easily explain the overallimaging function. As another example, most practical systems will usesignificantly larger arrays, particularly at the sensor array andpossibly also at the filter module. In addition, there need not be a 2:1relationship between the 6×6 regions at the sensor plane and theunderlying sensor elements in the sensor array. Each region couldcorrespond to multiple sensor elements, for example. As a final example,the regions labeled 1 in the object, 1A in the filtered image 255A and1A in the composite image 270 do not have to be exact images of eachother. In some designs, region 1A within image 270 may capture thefiltered energy approximately from region 1 in the object 250, but itmay not actually be an image of region 1. Thus, the energy collected bysensor elements in region 1A of image 270 may be integrating andsampling the image (or some transformation of the image) in region 1 inobject 250, rather than representing a geometrical reproduction of theobject at that region. In addition, effects such as parallax,vignetting, diffraction and optical propagation may affect any imageformation.

The approach shown in FIG. 2 has several advantages. First, multipleoptical images 270A-D are captured simultaneously at the sensor plane.Second, each captured image is filtered by a filter 227A-D within thefilter module 225, and each filter 227 may be designed to implementdifferent filtering functions. For convenience, the light distributionincident on the sensor array 280 will be referred to as a multi-filterplenoptic image 270, and the effect of the filter module may be referredto as filter-coding. In addition, since the filter module 225 is locatedat a conjugate plane SP′ rather than the actual sensor plane SP, andsince this typically means that the filter module will be much largercompared to what would be required at the sensor plane, the tolerancesand other mechanical requirements on the filter module are relaxed. Thismakes it easier to manipulate the filter module, compared to if thefilter module were located at the sensor plane (e.g., if attached to thesensor assembly).

Referring to FIG. 2a , note that the optical elements inside the dashedbox (excluding the filter module 225) form a conventional fundus camera.The image port IP′ of a conventional fundus camera typically is coupledto either an eyepiece to allow human viewing or to a detector array tocapture images. The conventional fundus camera can be converted to themultimode imaging system shown by adding the filter module 225 at thepupil plane SP′ and by coupling the image port IP′ to a plenoptic sensormodule (i.e., secondary imaging array 214 plus sensor array 280).However, to do this, the interior of the conventional fundus camera mustbe accessible in order to insert the filter module 225.

FIG. 3 is a diagram of an example of another multimode imaging system.In this example, the filter module 225 is positioned external to aconventional fundus camera. The conventional fundus camera itself is notmodified. Instead, additional relay optics create a conjugate to thepupil plane and the filter module 225 is positioned in this conjugateplane. The microlens array 214 is positioned at a conjugate to theprimary image plane IP and image port IP′. The sensor array 280 ispositioned with the same spacing relative to the microlens array 214, asin FIG. 2 a.

The following are some design considerations, using the system of FIG.2a as an example. This system must work within the anatomicalconstraints of the human eye. In general, the eye has a depth ofapproximately 22 mm, a pupil size of 2-8 mm, and an aqueous medium ofn=1.33. Accommodated at infinity, the f-number of the eye ranges fromapproximately 1.8 (at a pupil diameter of 8 mm) to 7.3 (at a pupildiameter of 2 mm).

Now assume a plenoptic sensor module with 100 μm pitch betweenmicrolenses, with 1 mm focal length in quartz with n=1.46. In thisdesign, the volume between the microlens and sensor array is quartz.This corresponds to an f-number of f/#=f/(nD)=1/(1.46*0.1)=7. A standardfundus camera typically provides 2.5-5× magnification. A fundus cameraimaging a fully dilated pupil would image at <4× magnification (7/1.8)in order to match the f-number of the lenslet array. For a partiallydilated pupil (5 mm), the camera would image at <2.5× magnification(7/2.9).

A standard fundus camera images a 30 degree field-of-view at 2.5×magnification. This 30 degrees corresponds to approximately 9 mmdiameter of the fundus. Thus, the image created by the fundus camera isapproximately 22.5 mm diameter. Further assume a sensor array 280 withsize of 24×36 mm. In that case, the entire image would fall onto thesensor area. Also assume that the microlens array sampling of the imagein this case is 225 lenslets across the image diameter. The lateralresolution may be limited by lenslet sampling. For example, a 9 mm imagesampled by 225 lenslets will result in Nyquist-limited resolution of 80μm. Desired fundus image resolution is 15 μm/pixel (30 μm lateralresolution), although this number varies widely by reporting agency.This resolution can be achieved by using more closely spaced lenses, orby processing the images to increase the overall resolution. However,magnification, sampling, f-number, and field-of-view can be additionallyadjusted using a secondary image relay between the fundus camera andplenoptic sensor module, as shown in FIG. 3. For example, if a dilatedpupil creates an f-number that is too small to match the lensletf-number, an aperture at a pupil plane can be reduced until thef-numbers match.

Depth resolution can be estimated as follows. Assume a fundus size of 9mm, 5 mm pupil, and 2.5× magnification. Assume a microlens array with100 μm lenslets on a sensor with 5.5 μm pixel pitch. Simulation shows anapproximate depth resolution of 0.25 mm. For eye imaging, a 0.25 mmdepth resolution should be sufficient to accurately measure funduscurvature.

In another wavefront sensing mode, a multimodal fundus camera is used toimage the anterior segment of the eye. In this mode, the plenopticcamera acts as a wavefront sensor that detects aberrations in theoptical wavefront passing through the anterior structures of the eye.Aberrations can be associated with anterior eye conditions, such ascorneal ulcers, cataracts, and refractive errors (i.e. myopia,hyperopia, and astigmatism). The illumination in this mode can betraditional fundus illumination, illumination from a point source, orillumination from collimated light.

In a multimode imaging system, it is also possible to include a viewfinder to enable the examiner to view an image through the view finderat the time of image capture. A beam splitter or a single lens reflexcan be used to split the optical path and direct the image to theplenoptic sensor module and to the view finder. For example, either asingle lens reflex or a beam splitter may be inserted at the relay plane(as shown in FIG. 2a or 3) to allow a medical expert to look at theretina, while the plenoptic image of the retina is captured on thesensor array of the same device.

In other embodiments, a multimode imaging system may include a set ofdifferent filter modules. Each filter module may be used for a differentpurpose. For example, one filter module may be used for spectralimaging, while another filter module may be used for depth imaging.Different filter modules can be inserted into the device.

The multimode imaging systems described can be designed and manufacturedas original instruments. Alternately, existing fundus cameras can bemodified to become multimode. In one embodiment, an after-marketconversion kit may be used to convert a conventional fundus camera to amultimode fundus camera. The conversion kit includes a plenoptic sensormodule with a microimaging array and a sensor array. The original funduscamera is equipped with a conventional sensor. During the conversion,the plenoptic sensor module replaces the conventional sensor, such thatthe microimaging array (e.g., a microlens array or a pinhole array) ispositioned at an image plane of the conventional fundus camera. Forexample, the microimaging array may be positioned at the plane where theconventional sensor was previously located.

Although the detailed description contains many specifics, these shouldnot be construed as limiting the scope of the invention but merely asillustrating different examples and aspects of the invention. It shouldbe appreciated that the scope of the invention includes otherembodiments not discussed in detail above. Various other modifications,changes and variations which will be apparent to those skilled in theart may be made in the arrangement, operation and details of the methodand apparatus of the present invention disclosed herein withoutdeparting from the spirit and scope of the invention as defined in theappended claims. Therefore, the scope of the invention should bedetermined by the appended claims and their legal equivalents.

What is claimed is:
 1. A multimode fundus imaging system comprising: afirst fundus imaging subsystem comprising an objective lens, the firstfundus imaging subsystem characterized by a field of view, magnificationand working distance to form an optical image of a fundus of an eye whenthe first fundus imaging subsystem is positioned in front of the eye,the optical image formed at an image plane of the multimode fundusimaging system; a filter module positioned at a pupil plane of the firstfundus imaging subsystem or at a conjugate thereof, the filter modulecomprising a plurality of filters; and a second imaging subsystemcomprising a microlens array and a sensor array, the microlens arraypositioned at the image plane of the multimode fundus imaging system,and the sensor array positioned at a conjugate of a position of thefilter module, the sensor array capturing a plenoptic image of thefundus of the eye, the plenoptic image containing images of the fundusof the eye filtered by each of the plurality of filters; wherein thesystem is operable in a depth imaging mode, in which the plenoptic imagecaptured by the sensor array is processed to provide a three-dimensionalimage of the inside of the eye; and wherein the system is switchablebetween (a) the depth imaging mode in which the plenoptic image capturedby the sensor array is processed to provide the three-dimensional imageof the inside of the eye, and (b) a multi-filter imaging mode in whichthe plenoptic image captured by the sensor array is processed to providetwo or more different filtered images of the inside of the eye.
 2. Themultimode fundus imaging system of claim 1 wherein the filter modulecomprises a plurality of different spectral filters.
 3. The multimodefundus imaging system of claim 2 wherein the different spectral filtersare selected to detect diabetic retinopathy.
 4. The multimode fundusimaging system of claim 2 wherein the different spectral filters areselected to detect oxy-hemoglobin.
 5. The multimode fundus imagingsystem of claim 4 wherein the filter module comprises a first spectralfilter at wavelengths for which the oxy/deoxy-hemoglobin spectra areseparated and a second spectral filter at a wavelength for an isosbesticpoint.
 6. The multimode fundus imaging system of claim 5 wherein thefirst spectral filter has a passband in the 600-700 nm range.
 7. Themultimode fundus imaging system of claim 5 wherein the second spectralfilter is a narrow bandpass filter centered around approximately 586 nm.8. The multimode fundus imaging system of claim 5 wherein the secondspectral filter is a narrow bandpass filter centered aroundapproximately 808 nm.
 9. The multimode fundus imaging system of claim 2wherein the different spectral filters include an infrared filter. 10.The multimode fundus imaging system of claim 1 wherein the filter modulecomprises a plurality of different polarization filters.
 11. Themultimode fundus imaging system of claim 10 wherein the differentpolarization filters are selected to detect the optical disc of the eye.12. The multimode fundus imaging system of claim 10 wherein thedifferent polarization filters are selected to detect glaucoma.
 13. Themultimode fundus imaging system of claim 1 wherein: the filter modulecomprises a plurality of different spectral filters and a clear filter,and the filter module is translatable relative to the first fundusimaging subsystem; and switching between the depth imaging mode and themulti-filter imaging mode comprises translating the filter module sothat the clear filter is illuminated when the system is used in thedepth imaging mode and the spectral filters are illuminated when thesystem is used in the multi-filter imaging mode.
 14. The multimodefundus imaging system of claim 1 wherein the system is further operablein (c) a wavefront sensing mode, in which the system acts as a wavefrontsensor to detect aberrations in the optical wavefront passing throughanterior structures of the eye.
 15. The multimode fundus imaging systemof claim 1 wherein the first fundus imaging subsystem is a conventionalfundus camera that produces a single two-dimensional optical image ofthe fundus of the eye, and the microlens array and sensor array are partof an after-market conversion kit used to convert the conventionalfundus camera to the multimode fundus imaging system.
 16. The multimodefundus imaging system of claim 15 wherein the filter module ispositioned within the conventional fundus camera at the pupil plane ofthe conventional fundus camera.
 17. The multimode fundus imaging systemof claim 15 further comprising relay optics, wherein the filter moduleis positioned external to the conventional fundus camera at a conjugateof the pupil plane of the conventional fundus camera.
 18. The multimodefundus imaging system of claim 1 wherein the first fundus imagingsubsystem is characterized by a field of view, magnification and workingdistance to form an optical image of approximately a 9 mm diameter ofthe fundus of the eye.